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; • AUDIO-BASED METHOD AMD APPARATUS FOR 

0 CONTROLLING OPERATION OF AN APPLICANCE 



5 Field of the Invention 

The present invention relates to methods and apparatus 
for controlling appliances and other devices, such as food 
processors, mixers and microwave ovens, and more particularly, to 
an audio-based method and apparatus for automatically detecting 
10 when a task performed by an appliance has been completed based on 
specified sound patterns and to automatically trigger a 
subsequent step (if any) or terminate the operation of an 
appliance. 



4-5 Background of the Invention 

iD The marketplace offers a number of consumer appliances, 

such as microwave ovens, food processors, mixers and blenders, 
Cm that provide an ever-growing number of features intended to 
F increase the convenience and capabilities of these appliances. 
fft£0 Most appliances, for example, have a one- touch feature that 
p allows the appliance to be activated in a desired mode with the 
:f7 press of only a single button. Thereafter, the appliance can 
;jj typically operate in the desired mode in an unattended manner 
until the human operator deactivates the appliance, allowing the 
25 human operator to perform other tasks while the appliance 
performs the desired primary task. 

While such features have greatly improved the 
convenience of many appliances, they still require the 
affirmative action of the user to terminate or deactivate the 
30 appliance or to adjust the configuration of the appliance for a 
subsequent task. Thus, even though the appliance may not require 
any direct human intervention during the performance of the 
primary task, the user must nonetheless pay attention to the 
operation of the appliance to determine when the appliance should 
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; - be manually terminated or adjusted for a subsequent step. For 
many tasks, such as the mixing of certain types of bread dough, 
the precise time at which the appliance is stopped or adjusted 
may be critical. Thus, if the user becomes distracted while 
5 performing another task, the intended result of the primary task 
may not be achieved. 

It has been observed that there is often a predictable 
relationship between certain sound activity of an appliance and a 
corresponding manner in which the settings of an appliance should 
10 be adjusted. For example, when certain ingredients are combined 
in a mixer, the mixing task is complete when the sound emanating 
from the mixer demonstrates a certain pitch or another 
predictable audio feature. Similarly, it is well known that 
O microwave popcorn is finished popping when there is an absence of 
^§.5 a popping sound for a specified interval and that water in kettle 
pi has boiled when the kettle begins to whistle. 

^ There is currently no mechanism, however, that can 

4p correlate the audio activity of an appliance to a corresponding 

adjustment to the appliance. A need therefore exists for an 
fr€0 appliance controller that monitors audio activity from an 
y, appliance and automatically adjusts the appliance in response to 
; I3 predefined audio activity. A further need exists for an 
;u appliance controller that can learn the correlation between the 

audio activity of an appliance and a corresponding adjustment to 
25 the appliance. 
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Summary of the Invention 

Generally, a method and apparatus are disclosed for 
monitoring audio activity associated with an appliance and 
automatically adjusting the appliance in response to predefined 
5 audio features. The disclosed appliance controller includes one 
or more microphones focused on one or more appliances. 

According to one aspect of the invention, a number of 
appliance adjustment rules define various audio features that 
suggest when an appliance setting should be adjusted. Each rule 
10 contains one or more specified audio features and a corresponding 
appliance adjustment that should be performed when the rule is 
satisfied. The detected audio feature may be a static audio 
characteristic, such as a specified pitch or volume, or a time 
o varying audio characteristic, such as a specified pitch or volume 
'%5 over an interval of time. 

CO The appliance adjustment rules may be predefined or 

, t P dynamically determined by observing user behavior during a 
4? learning phase. In a learning phase, a classifier learns 
^ distinguishing audio features when a user makes a manual 
f§0 adjustment. During operation of the appliance, the appliance is 
automatically adjusted when the observed audio features are again 
'Jf detected. 

1U A more complete understanding of the present invention, 

as well as further features and advantages of the present 
25 invention, will be obtained by reference to the following 
detailed description and drawings. 



Brief Description of the Drawings 

FIG. 1 illustrates an appliance controller in 
3 0 accordance with the present invention; 

FIG. 2 illustrates a sample table from the appliance 
adjustment rules database of FIG. 1 in accordance with the 
present invention; 
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" FIG. 3 illustrates an exemplary appliance learning 

process embodying principles of the present invention; and 

FIG. 4 is a flow chart describing an exemplary 
appliance monitoring process embodying principles of the present 

5 invention. 

Detailed Description 

FIG. 1 illustrates an appliance controller 100 in 
accordance with the present invention. As shown in FIG. 1, the 
10 appliance controller 100 includes one or more microphones 150-1 
through 150-N (hereinafter, collectively referred to as 
microphones 15 0) that are focused on one or more appliances 160, 
such as a microwave oven, food processor, mixer or blender. The 
Q audio information generated by the microphones 150 is processed 
Jts by the appliance controller 100, in a manner discussed below in 
ffl conjunction with FIG. 4, to identify one or more predefined audio 
events suggesting that the appliance (s) should be adjusted. In 
one implementation, the present invention employs an appliance 
^ adjustment rules database 200, discussed further below in 
f20 conjunction with FIG. 2, that identifies a number of audio 
|U characteristics that should initiate the adjustment of the 

appliance 160 in a specified manner. 
|M: As discussed further below, each appliance adjustment 

rule contains one or more audio features that must be observed in 
25 order for the rule to be triggered, and a corresponding appliance 
adjustment that should be performed by the appliance controller 
100 when the predefined criteria for initiating the appliance 
adjustment rule is satisfied. At least one of the criteria for 
each rule is an audio feature detected in the audio information 
30 generated by the microphones 150. The detected audio feature may 
be a static audio characteristic, such as a specified pitch or 
volume, or a time varying audio characteristic, such as a 
specified pitch or volume over an interval of time. Upon 
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; * detection of a predefined audio feature, the appliance controller 
100 automatically adjusts the appliance in a specified manner. 

As discussed below in conjunction with FIGS. 2 and 3, 
the exemplary appliance adjustment rules recorded in the audio 
5 adjustment rules database 200 may include a number of default 
rules that have been installed, for example, during a 
manufacturing or upgrade process, and optionally, additional 
rules that are dynamically learned by the appliance controller 
100 based on user activity. 
10 As shown in FIG. 1, and discussed further below in 

conjunction with FIGS. 3 and 4, the appliance controller 100 
includes an appliance learning process 3 00 and an appliance 
monitoring process 400. Generally, the appliance learning 
process 300 observes the operation of one or more appliances 160 
4 5 to learn a set of appliance adjustment rules that define when a 
ifg given appliance 160 should be adjusted based on observed audio 
^ characteristics. The appliance monitoring process 400 processes 
Jg the audio information obtained by the microphones 150 and detects 
one or more predefined actions that should trigger the adjustment 
110 of an appliance 160. 

Jf7 The appliance controller 100 may be embodied as any 

yy computing device, such as a personal computer or workstation, 
rf that contains a processor 120, such as a central processing unit 
(CPU), and memory 110, such as RAM and/or ROM. Alternatively, 
25 the appliance controller 100 may be embodied as an application 
specific integrated circuit (ASIC) (not shown) that is included, 
for example, in an appliance. 

FIG. 2 illustrates an exemplary table of the audio 
adjustment rules database 200 that records various rules for one 
30 or more appliances. Each rule in the audio adjustment rules 
database 200 identifies the corresponding appliance and includes 
predefined criteria specifying the conditions under which the 
rule should be initiated, as well as a corresponding action item 
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; • that should be triggered when the criteria associated with the 
rule is satisfied. Typically, the action item identifies the 
manner in which the appliance (s) 160 should be adjusted when the 
rule is triggered. 
5 As shown in FIG. 2, the audio adjustment rules database 

200 is comprised of a plurality of records, such as records 205- 
210, each associated with a different appliance adjustment rule. 
For each rule, the audio adjustment rules database 200 identifies 
the corresponding appliance in field 250 (and possibly a 

10 particular task performed by the appliance, such as mixing bread 
dough) , the corresponding rule criteria in field 2 60 and the 
corresponding action in field 270. 

For example, the exemplary appliance adjustment rule 

=!□ set forth in record 2 06 defines an audio characteristic (a 

%J5 specified sound pitch) that should trigger turning a mixer to a 

ffl high speed for a specified interval. Similarly, the exemplary 
appliance adjustment rule set forth in record 208 defines an 

45 audio characteristic (an absence of sound for a specified time 
interval when popcorn is popping) that should trigger turning off 

§£0 the microwave oven. 

jl^ As previously indicated, the appliance adjustment rules 

5 fi recorded in the audio adjustment rules database 2 00 may include a 
77 number of default rules that have been pre-installed, for 

example, during the manufacturing process, and optionally, 
25 additional rules that are dynamically learned by the appliance 

controller 100 based on user activity. 

FIG. 3 illustrates an exemplary appliance learning 

process 3 00 that may be employed in an exemplary embodiment to 

generate appliance adjustment rules. As previously indicated, 
30 the appliance learning process 300 observes the operation of one 

or more appliances 160 to learn a set of appliance adjustment 

rules that define when a given appliance 160 should be adjusted 

based on observed audio characteristics. 
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I - As shown in FIG. 3, the microphones 150 capture an 

audio signal that includes audio activity associated with the 
operation of an appliance 160. Thereafter, a window-based 
feature extraction is performed at stage 310 to generate audio 
5 feature intervals, such as 10 millisecond intervals. A 
classifier 330 processes the audio features to correlate audio 
activity with manual adjustments by a user during a learning 
phase. In this manner, the classifier 330 generates appliance 
adjustment rules that are recorded in the audio adjustment rules 
10 database 200. Generally, the classifier 330 learns 

distinguishing audio features when a user makes a manual 
adjustment that can later be applied to automatically make the 
same adjustment to an appliance 160 when the observed audio 

O features are again detected during operation of the appliance 

%5 160. 

ffl The classifier 33 0 may be embodied, for example, as a 

Bayesian classifier or a decision tree (DT) classifier, such as 
JS those described in United States Patent Application Serial No. 

^ , filed , entitled "CLASSIFIERS USING EI GEN NETWORKS FOR 

1*60 RECOGNITION AND CLASSIFICATION OF OBJECTS," (Attorney Docket No. 
3^ US010566) , assigned to the assignee of the present invention and 
€1 incorporated by reference herein. 

£7 FIG. 4 is a flow chart describing an exemplary 

appliance monitoring process 400. The appliance monitoring 

25 process 400 processes the audio information obtained by the 
microphones 15 0 and detects one or more predefined actions that 
should trigger the adjustment of an appliance 160. The exemplary 
appliance monitoring process 400 is a general process 
illustrating the broad concepts of the present invention, 

30 As shown in FIG. 4, the appliance monitoring process 

400 initially obtains one or more inputs from the microphones 15 0 
during step 410. Thereafter, the appliance monitoring process 
400 analyzes the audio information during step 420 using audio 
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I ' content analysis techniques. For a detailed discussion of 
suitable audio content analysis techniques, see, for example, 
Silvia Pfeiffer et al . , "Automatic Audio Content Analysis," Proc. 
ACM Multimedia 96, 21-40, Boston, MA. (Nov. 1996), or Dongge Li 
5 et al., "Classification of General Audio Data for Content-Based 
Retrieval," Pattern Recognition Letters 22, 533-44 (2001), each 
incorporated by reference herein. Generally, the audio content 
analysis is employed to recognize various features in the audio 
signals obtained by the microphones 150. 
10 A test is performed during step 430 to determine if the 

audio content analysis detects a predefined event, as defined in 
the audio adjustment rules database 200. If it is determined 
during step 43 0 that the audio content analysis does not detect a 
Q predefined event, then program control returns to step 420 to 
/fj.5 continue monitoring the audio information in the manner discussed 
C3 above . 

1- If, however, it is determined during step 430 that the 

Hp audio content analysis does detect a predefined audio event, then 

the audio event is processed during step 440 as indicated in 
H20 field 270 of the audio adjustment rules database 200 for the 
5^ identified appliance. Program control then terminates (or 
ilj returns to step 410 and continues monitoring user activities in 
-y. the manner discussed above) . 

It is to be understood that the embodiments and 
25 variations shown and described herein are merely illustrative of 

the principles of this invention and that various modifications 

may be implemented by those skilled in the art without departing 

from the scope and spirit of the invention. 
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